{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Navigation\n",
    "\n",
    "---\n",
    "\n",
    "You are welcome to use this coding environment to train your agent for the project.  Follow the instructions below to get started!\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "Run the next code cell to install a few packages.  This line will take a few minutes to run!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip -q install ./python #to run on udacity workspace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The environment is already saved in the Workspace and can be accessed at the file path provided below.  Please run the next code cell without making any changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: BananaBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 37\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: discrete\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "import time\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "\n",
    "#env = UnityEnvironment(file_name=\"/data/Banana_Linux_NoVis/Banana.x86_64\") #to run on udacity workspace\n",
    "env = UnityEnvironment(file_name=\"./Banana_Linux/Banana.x86_64\") #to run locally"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 1\n",
      "Number of actions: 4\n",
      "States look like: [ 1.          0.          0.          0.          0.84408134  0.          0.\n",
      "  1.          0.          0.0748472   0.          1.          0.          0.\n",
      "  0.25755     1.          0.          0.          0.          0.74177343\n",
      "  0.          1.          0.          0.          0.25854847  0.          0.\n",
      "  1.          0.          0.09355672  0.          1.          0.          0.\n",
      "  0.31969345  0.          0.        ]\n",
      "States have length: 37\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents in the environment\n",
    "print('Number of agents:', len(env_info.agents))\n",
    "\n",
    "# number of actions\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Number of actions:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "state = env_info.vector_observations[0]\n",
    "print('States look like:', state)\n",
    "state_size = len(state)\n",
    "print('States have length:', state_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Note that **in this coding environment, you will not be able to watch the agent while it is training**, and you should set `train_mode=True` to restart the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.0\n"
     ]
    }
   ],
   "source": [
    "env_info = env.reset(train_mode=False)[brain_name] # reset the environment\n",
    "state = env_info.vector_observations[0]            # get the current state\n",
    "score = 0                                          # initialize the score\n",
    "while True:\n",
    "    action = np.random.randint(action_size)        # select an action\n",
    "    env_info = env.step(action)[brain_name]        # send the action to the environment\n",
    "    next_state = env_info.vector_observations[0]   # get the next state\n",
    "    reward = env_info.rewards[0]                   # get the reward\n",
    "    done = env_info.local_done[0]                  # see if episode has finished\n",
    "    score += reward                                # update the score\n",
    "    state = next_state                             # roll over the state to next time step\n",
    "    if done:                                       # exit loop if episode finished\n",
    "        break\n",
    "    \n",
    "print(\"Score: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  A few **important notes**:\n",
    "- When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```\n",
    "- To structure your work, you're welcome to work directly in this Jupyter notebook, or you might like to start over with a new file!  You can see the list of files in the workspace by clicking on **_Jupyter_** in the top left corner of the notebook.\n",
    "- In this coding environment, you will not be able to watch the agent while it is training.  However, **_after training the agent_**, you can download the saved model weights to watch the agent on your own machine! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 100\tAverage Score: -0.03\n",
      "Episode 200\tAverage Score: 2.742\n",
      "Episode 300\tAverage Score: 6.36\n",
      "Episode 400\tAverage Score: 8.75\n",
      "Episode 500\tAverage Score: 12.05\n",
      "Episode 600\tAverage Score: 14.01\n",
      "Episode 700\tAverage Score: 15.20\n",
      "Episode 800\tAverage Score: 15.65\n",
      "Episode 900\tAverage Score: 16.08\n",
      "Episode 1000\tAverage Score: 15.60\n",
      "Episode 1100\tAverage Score: 15.92\n",
      "Episode 1200\tAverage Score: 16.57\n",
      "Episode 1300\tAverage Score: 16.78\n",
      "Episode 1400\tAverage Score: 16.74\n",
      "Episode 1500\tAverage Score: 16.81\n",
      "Episode 1600\tAverage Score: 16.73\n",
      "Episode 1700\tAverage Score: 17.11\n",
      "Episode 1800\tAverage Score: 17.14\n",
      "Episode 1900\tAverage Score: 15.93\n",
      "Episode 2000\tAverage Score: 16.42\n",
      "Training Time is 2462.7125222682953\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJztnXecFdXZx3/PFurSWZC+9KbSVhAFBBEBUbDECDGKLdYYzZuYoImoUSOx19hijyHmjUk0r4mIgiCK4II0kSYuvSxtWWD7nvePmbk7d+70O+3e+3w/H9h7z5yZ89wzM+c55znPeQ4JIcAwDMNkLllhC8AwDMOECysChmGYDIcVAcMwTIbDioBhGCbDYUXAMAyT4bAiYBiGyXBYETAMw2Q4rAgYhmEyHFYEDMMwGU5O2ALYoW3btqKgoCBsMRiGYVKKFStWHBBC5FvlSwlFUFBQgKKiorDFYBiGSSmIaJudfGwaYhiGyXBYETAMw2Q4rAgYhmEyHFYEDMMwGQ4rAoZhmAyHFQHDMEyGw4qAYRgmw2FFwCRFUfEhbNh7NGwxUoKqmjr8rWgHeHtYJmqkxIIyJrr84IWlAIDiOVNCliT6PLtgM55esAWNc7NxwaCOYYvDMDF4RMAwAVFyrBIAcLSiOmRJGCYeVgQMwzAZDisChmGYDMc3RUBEXYhoIRF9S0TfENFtcvq9RLSLiFbJ/87zSwaGYRjGGj8ni2sA/EIIsZKImgFYQUTz5WNPCCEe9bFshmEYxia+jQiEEHuEECvlz2UAvgXQya/yGCZVIJDp8Y++2YvKmtqkyth+8ATW7ix1de7X2w9j5+ETSZVvlyMnqvD5lgOuz/+q+BD2H62wlffj9ftQUV1fr3tLK1BUfMh12V6wblcpth08HqoMQEBzBERUAGAIgGVy0k+JaA0RvUpErQzOuZ6IioioqKSkJAgxGSZ0ln53ENe/tQKPfLgxqeuMeWQhLnh2iatzL/rjFxj1h4VJlW+Xma99hcv/tCyugXbCpS8sxeSnPrPMt2bnEVz3ZhHu+/f6WNqEJxbF3J/D4vxnluCsRz4NVQYgAEVARHkA3gVwuxDiKIDnAfQEMBjAHgCP6Z0nhHhJCFEohCjMz7fcYIdh0oIjJ6oAADsPl4csSTBs2lsGAKitc7/I7uDxKss8ZRU1ABDX+1bSGJ8VARHlQlICbwsh/gEAQoh9QohaIUQdgJcBDPdTBoZJRQR49bGXkGyNS0bhpDN+eg0RgFcAfCuEeFyV3kGV7SIA6/ySgWGY1IDMp02SJlsugKN76OOn19CZAK4AsJaIVslpdwGYQUSDAQgAxQBu8FEGhokMThohqwnldMPvBppkRVDHmkAX3xSBEGIJoPs0/8evMhkmXWDTkLdkyS0RKwJ9eGUxExiHj1dh3S53Lo1hsPS7g6iprQu0TL9NJGr2lJZjy/5jwRWoQ7lLbyEApi6uq3cciYvplJWljAiclXGiqgYrtvnvYnpAjkMVFqwImMC48I+f4/xn3Lk0Bs2KbYcx4+Uv8dj8TZ5fO8jG3oyRDy3AOY8vClsM1xi5uNbWCUx77nNc/dpXsbSs2ByBM03wy/9djUueX2p7rYJbJj6x2NfrW8GKgAmMbQeDWaTkBSVlUg8t6B5zplouvPzZivln1Y4jsbR605Cza63bJe21caIquQV+VthxgfUTVgQMo0NUeu2Mc/RuXZbLyWJFgaS7fmZFwDA6KI1J0D10VkDJo3fLyOWIgFyalFINVgQMowPFWuT0bgCigh8NrfqabucIMuUpYEXAMCZ42T45uVaad0B9Ra/uFEXgeGWxYhpK8/vBiiBNKauoxtaScF0Dzdh9pDw2IRsG1bV1+Ga3sSur1xaag8cqsbs0Pn5QXZ0wdafdvK8M5R5NUq7bVWrYCK7bVYoDxyqxPcTJfAFg7c5ST0cG6ivtL5O8ftyvI7B33v6yCizcsB/HK2uwYe9Rl2UFDyuCNOWHL36Jsx+LrmvgGXMW4LQHPw6t/EfmbcSUp5dgy/4y03xeNUvDHvgYn22OD7f8x0+34PxnluDr7YcT8lfW1GHCE4tx69yvky573a5SnP/MEjz1yWbd4+c/swSFD3yMMY8EE3FUjy+2HMQFzy7Ba58XJ30tvcV4V7yyXDrm8IZmOQxNMfzBT3D1619h4D3zMOnJz3x3O/UKVgRpyrd7Uqc3EgarZdfCkjJ9t70gJm3Xy/do95HExqKqRlrItmzrwaTKEEJgb6l0/Sgv5tt+SIoKunGvuWIOmmTnCI6mSIRTVgRMRkIxt0DzV9xPbxElnlC8uUKTlqRCqhOp4YmkWK28kNXsljm9m5T0HEFqTC6wImAyG4P31M/GkzQf4kUQ+nldkipuj4ri87LevfjpirJ2G/spRaqfFQGTmdS/4Ob4+R6buTQqKZRky6i+cpSVQlCiua0Dt/JFt8bjYUXAZCRW7WsQYaCzdM0O8coh2R5ynRApYRqqx19hgzcNpQasCBhd9pdVoEwVvREAig8cR51DP+yqmjrsOGTulvj9geA371Ze8MMnqrD94InEKKMeNAB6dahTBPaUVmBPqf7WlEdOVBu6kB6tqLZ0wVXLf+hEdWwrTCO0LqbbDh7HntJynKiqkeWpwiFVXBztvdtTWh4nb3lVLVbtOIIV2w5j39EKVNfqPw9Hy6V6OlZZE4vE6ea52H2kPG7/4xpNeXvkiXk9GfYdrcDxSv3JXcU0tONQ/bNi9FvizpOrs7KmFut3mztw7C2tiKu7ujoR2Mb2rAgYXYY/+AnGq9xPN+4tw9hHP8Xzi76Lpb23apflde55/xuMfnghSk/oN4iLN5Vg3KOf4l9fW1/LS5Qe/0//8jXGPLIQc/67QXNcIpmOoLYOE2SQtdEfPtyQsAG7ugG/6rXluuePeXihpQuu+jqrdxzB4N/NN81frVKIm/eV4axHPsXIhxbgB89Lm7wP/t18DL1fusYinXs38qEFcfLOfHU5Lnzuc1zy/BcY8ftPYs/DYU2QtRcXbwUA/Hv1bhQ+8DEWbtyPcY9+ivdX79aV0yg8+BlzFuCGt1bEvt/89kqMfrjeLbaqtg77jlbEpX28fh8AYMTvP8Elz38Rdz21++j+Mum8Bz74FgDwu3+vx+iHF+KgSQhpZe7jifmbcd7Tn5kGMTz9oU8w/eUvY9+fXrAZZz3yaSDrgVgRMIbsV/U2dx2Rej5FxfWx2dfutHZH/GxzCQCgtFxfESjugmG7Nn7xnb6bZrJ29f0mPXa1yeaIRlGqS132vX48fO05eiSzwc3OI/WjlPU67sibDO6dWt7lxfGyL9ooPQ/HDHreClbPhd7AVLlX6vI/kht5NQePxSuhb1Q99Q0a91X1PVI6M8ozrfy14yK6VHYDNhshAvVuzQDwpXyO4v7rJ6wIGFu4bQ9zs6VHrLpOvwfnh7eIHSznCAIQKCuAMlLVtl0f9M/+D7CbVascaw2eTe21jW6XmYzKofo5n2hO2LAiYBzh9EHOkWdEa2r1XxYlNYhGMWpkmfxkrzx86oRwNPGtLtbPO2KtiKW/RtWgN9KptVln2mzVJvNeeutNnHh0KecpZZrdc2MhXJzjEFYEjK/kKCMCA5tuXUg9JavygmgEzRporzryTq/j917JdhWclXuv3mUcB5STMduONCaHkL6ZCqWDImfsOXfzZAUwqmNFwPhKbrY8IjB4SWMLaIM2DdnM56dpRe83e+2u6PQ6bsp1I6qlInZRD24Dyhk9m3FyIHGTGicOBVE30bEiyHAOHKu05RLq9kFWTEOHjutPmsZsp5r00vJqHKuswZETVSgpq4QQApU1tYbeR1q2Hzyh6wp45EQV9pSWJzTC2k3UtcfN6kkIEXPjPF5ZE3O1VNBGodx5uFwuI74QIUTMddIqWFlljb2opHqboh85URWLZaRFbcIza6z3Ha2IuWkqz4adHrmSw8qNVeGwKl9FdW1ssrVCc78qqmttjwgqNb+9oro2zg1Xmcg+VlkTK2f3kfLYxL82/Mf3JcYunrE5AlXaoeNVOFZZYxrttaSssl6mADpJOf4XwUSVPaXlGPnQAvz8nD647ZzevpSRkyX1Na55vUj3uPLuaucIBt33Udz3ey4YgPnr9+GL7w6ieM4U0zJXbDuES2R3R21eI/dJI591AYHdR8pxxpwF+MWEPrh1fGI9vfXlNsx+7xvM//kYTHhiMbII2PpQfbmTnox3DX1mwRac3KlFgrJ5ftF3ePjDjQCA3RaeIjNf1Xcp1TL+sUU4s1ebuLTBv5uPs/rk6+a/4++rbV13xO8/SUh7ZN5Gy/OUhnHK00tM76OihP759S7MueQUNMzJxvjHFmHXkXIUz5mCqc9+Hpf/wuc+xzs3jLQl+4//tCzu+3urdmPu8h2x7yffMw/Fc6bg5HvmxdJufnul4fWue7MIz/5oCM4/taNhHqXDU1lTG3O/NSPOLZhNQ4yfKG5pCzbut32O486JxQl2J9EWbyoxdPHUYrVwxw5q2/Aei3pavEkKL71VViZ2OqYrtx9OqJpPN5TYlu/LrfoupXp8viWx3hZt0i9L7W7p9F5/9M1eR/nN5gvUZSuLrHap3Fm3axZybdhbZnuxo3b053Rjej2xi4oTQ4kDiXMu2tFIVGBFwPiKVWPiy2SxB9eKt1GbNzCe2fVT3HHKjtdOnPeNSXb1LTSz4TstXw+nnjzKb1CfZuT1pp0sjupcASuCDMbJM+n2+bVqk5VeoZX7qJPyvWhP9a5hdF2v2m+/9IArl0UXOJ2sNcutFtnI9TihfJdeQ4pnm130lh0Y1bEiUf1cQTQ1gW+KgIi6ENFCIvqWiL4hotvk9NZENJ+INst/W/klA+M96vbai0683Tj0Tt5xT0MZB6IuJfzynMoOSBN42dtVjxBrbCz4Apw9I2qyPaj4LIM6Vjo6mTwiqAHwCyFEfwCnA7iFiAYAmAXgEyFEbwCfyN+ZNEXPb1ptG1YaWqu2yskCK08Wp6kuYVW0Vw24X4vqglqsZ+cWqfOYzhGoRLbrDeTWNJTjUlGqlZXhqmODv04IQnf4pgiEEHuEECvlz2UAvgXQCcA0AG/I2d4AcKFfMjDeYachrq6tM1w4ZsTxSmmiLq4HaOMaNXLwMC21dSJOhvIqyeW0prbO0uXyRFUNSk9IbquKe+XR8hpUy6YJpUFy8zvtYLfB1otGqqTV1YkE10q3E5RacYziA5VXS/WlNg1VVNfq5ldP1No1DVXW1MW5mxpFY62sdjbpq5CdnVjvRmUo8lRU18ZFYa2s1q9jodEAdtxm3Zq4kiEQ91EiKgAwBMAyAO2FEHsASVkQUbsgZGC8R6sbBs6eh2aNcrDi7gm2z3v9i+KE471+81/LspQ8f/nJCJzRs20s/SdvFmHBhnrvnv6zPwQAXDSkE/5pEeF0wOx5CWnr9xzFDDki5Go5yN4p985Do9xsrJp9LgDt6lP37DxsHtIYkHzQte6HK7cfxsV//AKvXlWIxZsO6NapF6jdKdXMXb4Dc5fvQPvmDWNp/e7+UDdvmSpAm2l9qbTQ+U8vQZVK8Sr3VMvZJpFenWJUBiCtzTjl3nmxDgIgPceXDO2sk1uo/gdu++sqy7Ive2mpE1E9wffJYiLKA/AugNuFELb9+ojoeiIqIqKikhL7bnWMc7wyHFTV1uGgJrywV1YJI1u9NqyvWgmosVICTqiorouL/KlefeoEbd0UmywwUtBbILZym+S6+NnmA5i7fLtDKYxxGg7B6SDJbP5FXXKVD6MvNW564NU6E9hrdh3RySnhxLT5lYErqp/4qgiIKBeSEnhbCPEPOXkfEXWQj3cAoPvmCiFeEkIUCiEK8/P1F78wYWBtF43LrZPHlZ3U4KQoRXMMYiJQb+JXXQfhTkY69Bqy6T7qN37WWb37qH9leIGfXkME4BUA3wohHlcdeh/ATPnzTADv+SUDkz4YvaxeeHwkS5Ai5GYZv7JCuI+34wVOGztTRRDgogo/66x+iiDamsDPOYIzAVwBYC0RKYaxuwDMAfA3IroWwHYAl/ooA2OCk+ff9ToCC68h++XrnxOUj3xU0JvYVKd42ag5VXBOyzbLH+R99bOJjo0IormgOIZvikAIsQTG5ufxfpXLRAuvestGvc0IDAhUIZP97/VZr9T2XQTjsh0WbqYIgryvXilPrzo9YcAri5nAcfVqqE5SR86MxByBixATRu6GRsQifeoVHys/+UYnmep0HPLaTI5ATUP+lxF1hUBRFxAACgsLRVGRfvRKRp+CWR/EPo/tm4/Xrx4ed7ymti7mgjm4S0v865Yzdc9VIkROe3YJVu8sxejebVFSVpmwt6ua/GYNUVJWidvG98bK7Yfx2eYDlvJ2atk4LqhYVCmeMyVWP7ee3QvPLNiCSQNPwocOA6655aozCgzdQ68c2Q1vLt3mWVlvXzcCl2sidQZFz/ym+M4kvHMq8eY1w3HH31dj31Hj/avNePu6ETizV1vrjDoQ0QohRKFVPh4RZACfbkx0v3Xqkqf40B+tqDFVAgBicdRf+/x729dPBSWg5cXFWwHYD4HgBW996V1Db0WYY610UQIA8NH6vbbjJekRRF+dFUGGQg7dQN0Q/bFmcoQxmrazUToTPaJ+a1gRMP4hImLD9wkl5ESQE7RRb1CYRISI/hwBKwLGV9JXDdQrALebprvBrD3x3GspnW9ewGTsgjIm2rhtNJy0DRF/9j0jzEVcTGqQzDMShFsyK4IMJYi2K8iecphERQ94LkdEfleqI5DcveHJYsZz9pSWo2DWB/j4230Jx/aWVsS5jgLA3f9ah153/cdVWeXVtSg+mD7eHwAS6geIzojg7WXeBZwDgB+F5Dqabvxl2XbDEN52COLpCiQMNRMd1spuoHpRKtftKk1IS9ZdcZuNiJqpTlQUAZOedG/T1PcyeESQYXCT5T0ZYgFjQqJrmya+l8GKIENRd2LZOSQ5wthRimG8hBVBhsJNl3ewaYhJdVgRZBhK7z/qC1xSCR4QMKkOK4IMw6zN4vbMHaxUmVSHvYZSnGVbD+Kyl77EsrvGo33zRgCkzcy1aN0e3bZdq3YY78uaqSgB+RgmVeERQYrzxtJiAECRasPrv9jwJ+c+LMMwCqwI0hA7XkBszmCCZmjXlmGLwBjAiiANsRPwk9UAEzTpHIk21WFFwDBMIAS5IT3jDFYEaYid/V7ZMsQETZD7EDPOYEWQhjg1DfGQnQkEfswiCyuCiPCzuV/jd/9ebyvv3OXbMenJxXFp/1i5E+Mf+xSAPUWwWuUGWl5Vi6+KD+EnbxbZlpdhnMJ6ILrwOoKI8P7q3QCA2RcMsMx75z/WJqR9smG/6puzV27bweN4Yv4mR+cwjFN44BldeESQZrBbKBNVeI4gurAiSDPc6AFWHUwQ8IgguvimCIjoVSLaT0TrVGn3EtEuIlol/zvPr/IzlTohHL9wPIhgmMzGzxHB6wAm6aQ/IYQYLP9ztwciY4gAT8oxDOMM3xSBEGIxgEN+XZ/Rx9WIAIJHBQyTwYQxR/BTIlojm45ahVB+2rDzcOJ+wP9duxd//tLbTcwZxgt4jiC6BK0IngfQE8BgAHsAPGaUkYiuJ6IiIioqKSkJSr6U4o0vihPSbn9nlePrCMEvaaoxundbW/leumKY7Wu2zWvoVhxLrhvVHY1ysnWP5Wa7f/huGNMDk08+yfX5U07p4PrcdCJQRSCE2CeEqBVC1AF4GcBwk7wvCSEKhRCF+fn5wQmZQnhlzmGrUOrxmyn9LfMM794aQ7raH3S/fKV9paEwcWB7W/l+e77x+phLhnaOfb52VHdH5ffIb4pnfzTUVt7e7fIS0uwooRnDuziSKRUJVBEQkVr9XgRgnVFexhrPGnDWBGmL3+tKvFgboA5x4mdguiyXw95MCMHi28piIpoLYCyAtkS0E8A9AMYS0WBITU8xgBv8Kp+xj2BNkHLYaoBFauynrG5nnTa65EAV6V3aTnnprwZ8VARCiBk6ya/4VV4mkgkPKKOP3fbSiZIPS2dkxSkC5+fbPSdbZ7hh59QMGBDwyuJUJgU6e0yICAjfRwReNJJqk41jUxPZH0XomYbsVE8mhMZgRZAifLx+H+7/P3vRSZ1SXSvwxXcHfbk24w/2tiNNjdhTWUnMETjJ7nb+gUcETGS47s0ivLLk+7DFCJRfTeobtgih8dilg0yP222cOrVsbHisWaMc/HRcLydi+QKZmIaaNYy3Xj9xmXm9mJfjcrLYdYmpAyuCFCYFOnuu+ejnY3Dz2OQaqeI5U5KWo21eg6Sv4YZLhnU2PGbXb17AuPG79exeWHvvRPxyYr2yDavBMzO9LPvN+LjvFw0xrhfLcuRi3rpW32v9zsn9AAADOzbXnJf+qsC2IiCiUUR0tfw5n4icOfwyjAOi8+pFR5J47GxHatxT0Ds7rH6Fup3V2vG9tM9nWzToStkZ0O4nYEsRENE9AH4N4E45KRfAn/0SimEy8WW0i92RYBANuxf3iQw+27m+k9663XUEWuWTCc+i3RHBRQCmAjgOAEKI3QCa+SUUwzDm2NqXOkVMh8mtI3Bejrqh1xs1aUVgr6F6qoRUYwIAiKipfyIxDGOFLa8h0wtEp3FTN/4JjbCHYtofEfgnQ1Sxqwj+RkQvAmhJRD8B8DGkWEGMC8qranHnP9ag9ES1aT4hBB6ZtwErtulH8+YVwf5z4Fhl2CL4gldtmychJkyup/6uvzLYQTkWeWPvU8I8RfpjSxEIIR4F8HcA7wLoC2C2EOIZPwVLZ/5WtANzl+/AEx+bbxhfXSvw3MLvcMnzSwOSLLXo1qZJQtrUQR0xa3I/PDV9sOX5HVo0sp3XCrvRQL1C3Ytu31yKGtqnfR7evekMtGySCwB44ofxrpazzx+QVKTPIPjhafFeQeo2+c/XjgAgeffMPn8ALhjUEZNPlsKXFXYzD67XIDvLWhHIeiCLgLevG6ErQxA8edlgnNq5RaBlWioCIsomoo+FEPOFEHcIIX4phJgfhHDpimKXTIXFPlHmycviG/BTOrXA0zOG4MazemLa4E4AgAsGdTQ8/87z+sfyFugoFSf8aWYh/n7jyLi04jlTUDxnCn42vrfl+cMsGjIjWjXJRYMc6TV+avoQDOvWCqtmn4viOVPQIz8+2uY1o7rjpiRcct+75cy47+2aNfSmu6y6RtOGOXj1qkK9QzhFbhxvOKsnrhnVHc/MGILGDaTQ1n+/6Qx0aNEolrd4zhSsnn0uAGm9xKYHJzsS58xe9YpdrXQvld16FVdTAMjxKFLeqtkTUDxnCi4c0glPTR/iyTXtYqkIhBC1AE4QUbAqKo3h5t8btBOLTk1lXiriLKKkeo51DmQREKbmFDu4kVUvVo/fJOXDrznVqoqVw9oySTdPYpqXBF3TdoPOVQBYS0TzIXsOAYAQ4me+SJUhZMJCFT/R1p7eix5UDUuKwH1pTmMC6a3GtVV8EsoviMc1YaLW5JgWNz9N75SEctSNvjKNYOF5lCxBNw12FcEH8j/GQ6weIKsebqZblrQvi9P68LL+ssh6wZK5LA5GBKqsRO6ma92cped143d7lYxrqRdlAvqNvv8jgmA1gS1FIIR4g4gaAOgjJ20UQpi7vDBJk+kNvRXal8VpdakVbbKNDBEZuyfauJG1DocE6t8eWxGbZONhJaZfpqF4zyDtYi77ZboKYa36rNfb115Xr4r8eE8jOSIgorEA3oC0mQwB6EJEM4UQi/0TLTO4///Wo0mDxL1chRB46pPNpud+s7sUXxUf9ku0ULHzcmlfloY57kNneTLnaaQHbJybjGnIifDJtFl+TRHYndtxXbztldgxTWBYrjKX43a3M9PyQ+z42X1zHgNwrhDiLCHEGAATATzhn1jpjXLDDx6vwitLvsczC7Yk5Nl64Die//Q73fPz5IiM6aoEzLhlXE/DY7+a6CxaqdmLN21wR5zdr52jxk/dOPzWZE9hPS8ixeQwcWB73aireaoonL+aVO+xkqUyDRm1TT8/pw/ukOvmitO7YXCXlpgxInEfXrvhHPIa5mBMn3w8M8OdZ0uPtu7Wozpte5s1zMG4vvl44QppL+ZYj9+ih2+2oOzn5/RBYbdWuGRoZzTI9iZmZ9u8hhjbNx8tGufqlhkEdn9JrhBio/JFCLEJUrwhJgnMOgBmpoL+HaIT3aPfScayPHjRybauUTxnCu46r591RgB3TOyH7nJDon5Z+p3UDGf0SvTld/tC3XVef7x61Wlo1sj+Y56lepuuOqPAMN//TOiTsCG6cr9/cW5f3airt55dn9ZLtQl7brb1JPVt5/TGLXK46XbNG+Fft5yJds3qXS0Hd2lper6CUkoWAW9eMxwjerQxzf/wD07VTb/rvP5x4aXjTEO2JLFHVhbhtauHx7mCmqHIYTZHUNC2Kf5+0xlo0SQXr151miN5/nr96ZhySoeE9OHdW+H1q4cjS9XrCNqRxO5kcRERvQLgLfn75QBW+CNSBmGiCZy4E0YVP4bPQH3vOdnrx1WxgTnAySSuWh4re7pW0dc6/E3K85GTTSpZbQrqEt3JYjN5zeRxZc/3/nnSu6LdoHNePd51dXoyBItdRXATgFsA/AySjIsB/NEvoTIFM9uo2UudKkGwgvQo8asRdHLZ+L131Z4miXm1A766OkUR2CurRs6fm1W/YjbZkCNWdRiI+6hJGX6UL+I+J3oEAf4/x7U6FR/JyWI531NCiMcBabUxgIa+SZUhmL14PCKwxleFGOtmOzjFwe/V3l+lMTAaSWgvXVNbPyJQ6tnvR8ZpfadqLKyE2+jzc1ynYwYOurNnd47gEwDqPe8aQwo8xySB2YubKnrArPHzu1cTtfV4ThSf9uVXzAN2r1FdK52Qo5qYCOOZMZPWjTxBNIBGJdTHGorP4feCar1OX1RHBI2EEMeUL0KIY0SUXHCWNOfw8Sq8U7QDN4zpgTU7S7HrSDnOkyeKlNv+4Td7E86b+uwSbNxbhuHdW+ted9WOI1herB+NNGo46SG7azSSy2Nuwnaz4MqonMSSEkxDFiMCLTHTUDahqta4nKjiahFcSIrfS+Wkd49qQ1wRr2B3RHCciIYqX4ioEEC5PyKlB79+dw3m/HcDvio+jGnPfY6b315p67w1O0tRWVPnVmhrAAAdQklEQVSHzzYf0D1+85+DnaPPa2jeV1A/sNo2jCB5z2j3gAWAG87qAQCYc/EpptdX3DDzGuZgwoD28dcnxNKsGsHrRnWPC0qm5b6pA9G9bdOEzdKtmtYfDOuM03tISrtDC2nQrK0HPSV35chucR5Xv7/oFPRql4f8ZvoW16mDOsV979M+D93aNMFd5/UPbLI4r1His2A1V3zz2J6xQG3JYrdBvqww0TXWiBvGJLojJy5qk/420/n9gBTs0C63jOuF5o1y0LxRDl676jR0atkYPz9HJyhhREcEtwP4XyLaDen+dgRwmW9SpQFlFTUAgJpaHZeAJNhdWuHp9axYd99EAEDBLP0II+q2592bzsBFf/wi9j0rC7h36sCE8/ud1Ax3Tu6POycb+9orXDe6B64b3cPgKOGX5/bF/PX7LBvBgZ2a47fnD4iTQ+0RNLp3Phb+ciwKH5iPssp6xWLlNTRjeNdY5NAGOVkonjPF8jcBQGFBa3x4+5iYPOP6tcO4fu0M85/UolHctZs0yMGiO8YBqG+o3CoCvca8eM6UuLq6cmQ3yxAaN57VEy8sil/7ol7z4FSWcX3zAUijJCcrr2/Ta1gNGKDTSUmYIpD/zhxZoHsNIwWhd92BHVtgzb0TY2mfzzrbIG+E5giI6DQiOkkI8RWAfgDeAVAD4EMA3wcgX8qSSsN0r6jRvKy+TxaTH+aC+Aumwl30ymvItAwHcih4NULRi+/jJ0a7pAX5TkdtQdmLAKrkzyMB3AXgOQCHAbzko1wpT+wliNiEpteof161x6MfJ2V7/orKF4yCS6UVSu8x7MlibVWYNZxuFkwlW9VuQ1mQhVeWL7GGvL+kKVaKIFsIocxMXgbgJSHEu0KIuwGY7nBBRK8S0X4iWqdKa01E84los/zX3W4cKYTeEC9dN6Sp0cx6+b060knkTbtVHoWG3Sn1PdbUJPA6tyjP6LkNsn6DXllsqQiISDGAjQewQHXMyjD2OoBJmrRZAD4RQvSG5JI6y6acKUeqvpROUYdVqNEskUzG7c6s/pzUrdkLZacML8wBfj8L9T3WcJ86L0xD5gvK/G0cYyaohHKV474WH19mcEUBsFYEcwEsIqL3IHkJfQYARNQLQKnZiXJkUq2f4zRIUUwh/73QqcApg/zQfLppfyzpsheXYt43e9N2Q5rqWvdzBG7esTjTkEdvaYJ5w8o0ZOOa/i/0ksvxswwb9zLZsOBRwSjWkLZT4OdbHKk5AiHEgwB+Aal3P0rUv21ZAG51UV57IcQe+dp7ABi6SRDR9URURERFJSUlLooKF+WheXHR1ljasu8P4Ya3VoTec/OS28f3iX3W7rt7WoH+Wgg72H0POrdqggbZWbHomlquHNkNQP0etKMsApBpe3/3TzMPnKcOAOcX2gB1WrzssY7tm2/o7tm0YTaaNczB7AsG6h6/cEhH02B7atQN3UVD4l1jB3eRnqOfjJG8xR686BQ0b5RjOcK887z+aJybjTZ5DXSPK4H3BnbQd/ccKj+/Wi81CkLTaojcxjRCiC910jb5I05cGS9BnpAuLCxMuZYzldr6dfdNxMn3zLOd/yeju+PlzySnsdO6t45za7TrPmmF3epr3CDbdGPyIV1bxcn05+tGGLrCAom9vx+e1gW/endN7Hh+s4Z4avpg/OjlZRjRvbWj6KRueehi/SieCvXrCJJ/6F6/erjhsZzsLKy9b2JcmjJSePyHg9CrXTPcO3Ugqmvr8Pay7bZegpV3T0DrpvUNN4HQummDuHs2Y3hXzBje1fJaUwd1xNRBHQ2Pj+6dr/t89m6Xh837j6FtXkPd4yHogcBtQ94E1LbPPiLqAADy3/0W+VOWFNIDSZHSVq4kQ3zY/e1+ux3G5giSvpJH5jUb9RKlx8bqV9ePuDLXfdRr3gcwU/48E8B7AZcfGKlk/nH6zKntxV49r1GpLqsXUAhETssHtbLYDL2yXc37hKAhjCaJFcJwz43aZLFriGgugKUA+hLRTiK6FsAcABOIaDOACfJ3JmSSefnSdeLbzjsflXDgWSF6DenVQJjrGtwQW/JjcDuVdKfbiSZDVDemcYwQYobBofF+lckEg9kiolTHlheQ04v67j8aTDFmqMt20oZplVcoz1NMBPPSA11ZHFhJEkGbhlKej9fvw8FjlZb5UqQzBMBFz1aVPZUHBOYbAxkdq0+3P0fgL0GYhgx/q1nQOROBojiSNB4RhGAaSvM5gpTmRFUNrnuzCDNfW26ZN1WGxUDiQ/frSf0wtm8+Tu6UGJALQNy+q3bXCswY3hUNcqTHTYk8akbHlo0Nj/lZt8rG8m2a6kcBvW1878jd22tGdQcgRSR1g9EdVIdCd9JZuFB2Bx3dJ98yb1hVeWrnFhgrB7WzkkHJN3WwvkeS7dAVDlr3yLmPMvUoQdW2HThhmTdKbUXzRjk4KkdDtcNNY3viprGJ4XkVTu1sb7NzNQ9dfAoeMgk5rbxMN5zVw1ZUUiD5XpNegz59eFdMN3BV3Pr785CVRVgihwiPSqd24sCTPHPbVfO3G0biT59txQMffGuZV937H6px2XVCUCOF9386KvbZarK4Z36e7d+j5NNzUXYyh8MjgnQhQt1Gf1ec+njxiBFGFMqoE5UJ82SonyxO/d/iFlYELki2GYiQjgCQpNdQGjQEdtHaitPlt4fxOLqYcvCN+vvpEB+F5RFBhHFybyLW1vtGBnei7E8WR03zu8Cv+xylqonSsxypjWkY90TpAbcimYcuQu+OY9zeIqfnRf1ZiNI9DGVBWUDdNkeTxTwiSG1WbDuEkrJKrNttGpw1WFJgc5V0M7ekC3aVWMR1nSlRfPZ4HUGKc8nzSzH12SWh9wLVG6NbkcxD5/d2lHooES7b5Om7ePqNU1PPhAHtAQA985smHPNqY3e/MbrLFwySXImHdnW2x5QS4VPZ7/eWccZean5z9ZmS+602aulZffLRINteE6mNuqrc1wtVLqfd2ybefyPSZmVxOmPVEOyx2GDe76Go4sJWWl6NQfd95GsZYXDNqO4x33k3TD+tC/761Y6k5bD7so7o0cawvh65dBAeuXRQ0rL4hdWzOrZvO1fPgtZF+Y6J/XDHRGcb3XvFtaO641qd5+mNa4wjsaoRArh36kDcO7U+PLf6vj45fYhjmXhEEGFSbfhrdwvDTHWbcztqS7XngEk9eI4gwsRsiUnepaBskXZLyTQ14NVLlm71FrY5M1Xxo9GO2p7FjBqPXpSgFySlkvtiSgxOUqc6k6a+8xOuHFEmhV4vQ1gROCBdV5TyS+4OrjcmXWBF4AJtD/ur4kOoqa2zff7v/7PBa5GSImPnCFwq9nTtEDCZCysCB+gNAVduP4xLX1iKJz/eHLxAFjTKzQYAXH56N8+uefkI671jkyFasydBXMWcEaoIoGFwmlz+6N7WkUStmCxHre3tMkpqJjL9tC6BlMPuow7Q6wceKJP2JtiwtyxUW/wDF56M3/5rXVxabnYWNj84GTlZhF9P6ocdh05g7KOfxo5fN6o7Zk125rL3wIUneyFuJHDtNRTQbd7y4ORA1mmYjQiHdm2FTQ9MjoUQT4YfDOuMqYM6enKtTGDzg5ORHdBonRWBA/QaevU2gWFOGhktfMmV07MJaJgbnycri5Bjc8GMQqaakdR45T1mhdN74xarDoyXDXc6KQG/vf9yA7r/AJuGHCE0fwH1fqYCdSFqAjd2a27Sk4Prj0kXWBEkiTIiqBPR9yqMsptbkGa1pDe08UYMhokMrAgcoNdWqVfvhjoiSJPWKUjLU7JVxlYyBkiPjgErAgco5pcTVbXSdyGwctthAMCq7Yfx3f7joclmh4QHNkMbsmR/diot0LMDz/swrAicoHn/v95xBE8v2AIAOFpRg/Oe/iwEoSRSvWkaIkevLOwWrrukHXrkS+6PZ/VtF7Ik3nBOfyk6aoeWjUOWhAkL9hpKgsPHq8IWIYadTqpZT3btvefilHv9iVRqhzN7tcXXd09Aq6YNrDOHTK92efj67glo2SQ3bFE84cazemDG8C5o2ST6dR8l0mkgxYrAAdpmtC7FuuFaPaB2f2vWKPxGLXAlkISJJxUUll2IiJWAC9LJQsimIQdob3yYk8NaXLmPplGPxgmZ+rsZxohQRgREVAygDEAtgBohRGEYcjhF29hGSA/YInFEkNmk2O1jGN8I0zQ0TghxIMTyHaNtSKPkPRIhURiGSTHYNGSToxXVOHyifnJ4074yfLn1YIgSxWNHD3DUTIbxjnQyMYalCASAj4hoBRFdr5eBiK4noiIiKiopKQlYvESG/G4+pjy9JPb93CcW442l2wKVQXHza9IgO+FY3/bWm9U3lyeEu7ZuAgA4tXMLD6VLHZSN1nu3s64zhjGiQ4tGAIAzerYJWZLkCcs0dKYQYjcRtQMwn4g2CCEWqzMIIV4C8BIAFBYWht6VrfXJRejHp3fFn7/cbivvc5cPQemJajRukI2aWoGH523E3OXSucNthCtu1bQBlt55NvLzGmJPaQW6yArBDqtnn5s2kwoXD+2M0wpaO/r9DKOlW5um+HzW2ejQvFHYoiRNKIpACLFb/rufiP4JYDiAxeZnpSendm4JwJ4iaJiTjXbN60cDzRs5v30dWkiLhpw2gi3SxGdegZUA4wWd0mQRXuCmISJqSkTNlM8AzgWwzvwshmEYxi/CGBG0B/BPOb5JDoC/CCE+DEGOaJCMxSlNTDUMw4RL4IpACLEVwKCgy40qyXjy+L0xBsMwmQG7j9rgUIRiCjEMw3gNKwILlm09iKH3zw9bDF289mNW3OGYePrwZutMmsNB5yxYs7PU82sWtGmCpg1z8M3uo4YrgmcM74K5y3d4XrYZH94+BmUV1YGWmQq8e9MZOHKC64VJX3hEEAK92zdDQZumAIznisfZiHWf5fGIoEXjXHRuxW6VWpo1ymV3UyatYUUQFh404jxZzDCMF7AiCAmrJpy3D2QYJihYEYREltzQJxM1lHUFwzBewIpAprZOoLq2LrDyuBFnGCYqsCKQ+fGflqH3b/4bWHmKHjBaUOZGT7RKs3hADMMEA7uPyiz1eW+BK0d2w5uqsNVWcwB2RgzaLB//z1m8+I1hGMfwiMBH1O6dUwd1jDvmh2WoTV5D9LaxLwHDMIwaVgQBoe3hk8Vksa05BJ5oYBjGA1gReEy2ahhgZv5RDiWz7zGrAYZhvIAVgcdkGzb+FPdJ+ZbMxmc8IGAYxgtYEWioS3JLSqPGOdE0JP01GhHwqmGGYYIirRXBK0u+R8GsD3Csskb3+HurdqFg1gcYMLt+X5zqOmktwaZ9ZSiY9QEe/M+3jso8uVP9hvD9O9RP3CobxwNSfCFlA/n2Sex3ytFCGYbxgrRWBK9/8T0A4NAxfZfKvxVJ0T1PVNXG0mpqpR76su8POS7vxrN64tWZp8W+v/DjYbHPvdrl4Zz+UiC5nCzCTWN74U9XFmLSySfhnetPx6zJ/Qyvu+TX4/B/t45KSP9hYRfHMjIMw2hJa0Ugd+4NzTVKo68mmdXFsyb3i9vkvXnj+AVeo3vnAwBysrOQnUU4Z0B7EBFG9GiDlo2NF4N1btUkbqShwPGIGIbxgrRWBApG7WWdjn2+Wkc5uCVLU7DyNcdO/Ghu4xmGCYi0VgRKQ2/Uc67RmRj2Mt6QtlTluy1FwDAMExBprQiUDr9Rs1urowgUc1Ey/v0KCfpHTsjJtlYErCoYhgmK9FYEckC3+pDPItbACyF0zUBV8oigqib5kYHWBVQpOyfLutrZ/s8wTFCkddA5pcN/xSvLsHn/MVvnnPP4Is/K17bliuLJ1jENafPmNcz2TA6GYRgz0loRKNYdu0rADvdPG4h/r9mD5SbupX+9/nRdH/8aebSRa2Ia6tG2KX4+oQ+GdWudvLAMwzA2SGvTkPHW8O7p0roJLh7SyTTP6T3aoFubpgm9fGVyOtvENDSsWytcoIlUyjAM4ydprQj0JoOTRcC+etHOESgT0XojAg4pwTBMWISiCIhoEhFtJKItRDTLr3L03EOTpbZW2N5nWDsiqJVXuOnNESh4LzHDMIw5gSsCIsoG8ByAyQAGAJhBRAP8KMuPEYET5aJdUFZdp4wI0nogxjBMihFGizQcwBYhxFYhRBWAvwKY5kdBvowIHFxT2+9XzjVbUMYGIoZhgiYMRdAJwA7V951ymufUeLhKWCE7i9DUpmunMiBQPIiUVct6pqG8RpIDV0sXG9DzkgOGYZIhDPdRvWYroZtNRNcDuB4Aunbt6qqgM3q2xZItB1ydq8eUUztgbN98NMjOQklZJR74QApR/d4tZ8ZFMFUgIjwzYwiGdmsFoH5EoGcamjTwJNw/bSAuVUUUffnKQvTMb2oq04tXDENf3qeYYZgkCGNEsBOAOn5yZwC7tZmEEC8JIQqFEIX5+fmuCpo4sL07CQ148MKT0Sg3G1lZhCtHFsTSB3VpiZE92+iec8GgjujUsjEA8wVlWVmEK0YWoFFu/WhjwoD26JGfZyrTxIEnoaCtubJgGIYxIwxF8BWA3kTUnYgaAJgO4H0/CvJ6ikDt4unGHKN4DZktKGMYhgmawE1DQogaIvopgHkAsgG8KoT4xo+y9MJMuyEni1BTJ0AqtemmKa+ptV5QxjAMEzShhJgQQvwHwH/8L8eb62RlEVAn4txB3QSFq6kzXlDGMAwTFmndNfXKMuTV/gE1NhaUMQzDBE16KwKPhgRt8xomXM9NU944VxqANWnAkUUZhokOaR19VLv4q3mjHBytqDHMP3NkNxw4VoUP1u4BAPzhklPQKDcbw7q1wsKNJWjWqN7H381k8ewLBqBXuzyM7dPO+ckMwzA+kdYjAu3K4ud/PCz2uaBNEwDAo5cOAgAM7doS9007Gc9dPhQDOjQHAAzs2ALTBndC51ZNcMXp3eKu5WaOoEXjXNw0tqc058AwDBMR0loRaEcEeiEn6uridzED6r2NtLGCGIZh0pG0VgTahl/x449LUxp9VS9dmQpgL0+GYTKBtG7qtA2/XughZdSQrTMi4D0CGIbJBNJaEdTUWo8IlEZf7dKpnMWmfIZhMoG0VgSJ/vr135WYPkoedYyfhjlStfCkLsMwmUBau4/ePK4XauoEWjTORbc2TXBO/3bod1IzXDGyG8b0zsc/v96F6ad1xa7D5bhmVPfYeS9eMQzvrtiFHhbB3O4+fwDO7KUfbI5hGCZVIK8WXflJYWGhKCoqClsMhmGYlIKIVgghCq3ypbVpiGEYhrGGFQHDMEyGw4qAYRgmw2FFwDAMk+GwImAYhslwWBEwDMNkOKwIGIZhMhxWBAzDMBlOSiwoI6ISANtcnt4WwAEPxfEKlssZLJczoioXEF3Z0lGubkKIfKtMKaEIkoGIiuysrAsalssZLJczoioXEF3ZMlkuNg0xDMNkOKwIGIZhMpxMUAQvhS2AASyXM1guZ0RVLiC6smWsXGk/R8AwDMOYkwkjAoZhGMaEtFYERDSJiDYS0RYimhVguV2IaCERfUtE3xDRbXL6vUS0i4hWyf/OU51zpyznRiKa6LN8xUS0VpahSE5rTUTziWiz/LeVnE5E9LQs2xoiGuqTTH1V9bKKiI4S0e1h1BkRvUpE+4lonSrNcf0Q0Uw5/2YimumTXI8Q0Qa57H8SUUs5vYCIylX19oLqnGHy/d8iy57UVnwGcjm+b16/rwZyvaOSqZiIVsnpQdaXUfsQ3jMmhEjLfwCyAXwHoAeABgBWAxgQUNkdAAyVPzcDsAnAAAD3AvilTv4BsnwNAXSX5c72Ub5iAG01aQ8DmCV/ngXgD/Ln8wD8F9I+n6cDWBbQvdsLoFsYdQZgDIChANa5rR8ArQFslf+2kj+38kGucwHkyJ//oJKrQJ1Pc53lAEbKMv8XwGQf5HJ03/x4X/Xk0hx/DMDsEOrLqH0I7RlL5xHBcABbhBBbhRBVAP4KYFoQBQsh9gghVsqfywB8C6CTySnTAPxVCFEphPgewBZI8gfJNABvyJ/fAHChKv1NIfElgJZE1MFnWcYD+E4IYbaI0Lc6E0IsBnBIpzwn9TMRwHwhxCEhxGEA8wFM8louIcRHQoga+euXADqbXUOWrbkQYqmQWpM3Vb/FM7lMMLpvnr+vZnLJvfofAphrdg2f6suofQjtGUtnRdAJwA7V950wb4x9gYgKAAwBsExO+qk8vHtVGfoheFkFgI+IaAURXS+ntRdC7AGkBxVAu5BkA4DpiH9Bo1BnTusnjHq7BlLPUaE7EX1NRIuIaLSc1kmWJQi5nNy3oOtrNIB9QojNqrTA60vTPoT2jKWzItCz4wXqIkVEeQDeBXC7EOIogOcB9AQwGMAeSENTIHhZzxRCDAUwGcAtRDTGJG+gshFRAwBTAfyvnBSVOjPCSI6g6+03AGoAvC0n7QHQVQgxBMD/APgLETUPUC6n9y3o+zkD8Z2NwOtLp30wzGogg2eypbMi2Amgi+p7ZwC7gyqciHIh3eS3hRD/AAAhxD4hRK0Qog7Ay6g3ZQQqqxBit/x3P4B/ynLsU0w+8t/9YcgGSTmtFELsk2WMRJ3Bef0EJp88SXg+gMtl8wVk08tB+fMKSPb3PrJcavORL3K5uG9B1lcOgIsBvKOSN9D60msfEOIzls6K4CsAvYmou9zLnA7g/SAKlu2PrwD4VgjxuCpdbVu/CIDizfA+gOlE1JCIugPoDWmCyg/ZmhJRM+UzpMnGdbIMitfBTADvqWS7UvZcOB1AqTJ89Ym4nloU6kxVnpP6mQfgXCJqJZtFzpXTPIWIJgH4NYCpQogTqvR8IsqWP/eAVD9bZdnKiOh0+Tm9UvVbvJTL6X0L8n09B8AGIUTM5BNkfRm1DwjzGUtm9jvq/yDNtm+CpN1/E2C5oyAN0dYAWCX/Ow/AWwDWyunvA+igOuc3spwbkaRXgoVsPSB5ZKwG8I1SLwDaAPgEwGb5b2s5nQA8J8u2FkChj7I1AXAQQAtVWuB1BkkR7QFQDanXda2b+oFks98i/7vaJ7m2QLITK8/ZC3LeS+T7uxrASgAXqK5TCKlh/g7As5AXlnosl+P75vX7qieXnP46gBs1eYOsL6P2IbRnjFcWMwzDZDjpbBpiGIZhbMCKgGEYJsNhRcAwDJPhsCJgGIbJcFgRMAzDZDisCJi0hohqKT6qqWlUSyK6kYiu9KDcYiJq6+K8iSRF7mxFRP9JVg6GsUNO2AIwjM+UCyEG280shHjBOpevjAawEFLkzM9DloXJEFgRMBkJERVDCjEwTk76kRBiCxHdC+CYEOJRIvoZgBshxfBZL4SYTkStAbwKaWHeCQDXCyHWEFEbSAuY8iGtlCVVWT8G8DNI4ZWXAbhZCFGrkecyAHfK150GoD2Ao0Q0Qggx1Y86YBgFNg0x6U5jjWnoMtWxo0KI4ZBWiz6pc+4sAEOEEKdCUggAcB+Ar+W0uyCFJQaAewAsEVLQsvcBdAUAIuoP4DJIgf4GA6gFcLm2ICHEO6iPnX8KpJWsQ1gJMEHAIwIm3TEzDc1V/X1C5/gaAG8T0b8A/EtOGwUpHAGEEAuIqA0RtYBkyrlYTv+AiA7L+ccDGAbgKynEDBqjPpiYlt6QwggAQBMhxapnGN9hRcBkMsLgs8IUSA38VAB3E9FAmIf+1bsGAXhDCHGnmSAkbRnaFkAOEa0H0IGkbRRvFUJ8Zv4zGCY52DTEZDKXqf4uVR8goiwAXYQQCwH8CkBLAHkAFkM27RDRWAAHhBRLXp0+GdLWgYAUPOwHRNROPtaaiLppBRFCFAL4ANL8wMOQgq4NZiXABAGPCJh0p7Hcs1b4UAihuJA2JKJlkDpEMzTnZQP4s2z2IQBPCCGOyJPJrxHRGkiTxUrY4PsAzCWilQAWAdgOAEKI9UT0W0g7wmVBioR5CwC9bTiHQppUvhnA4zrHGcYXOPook5HIXkOFQogDYcvCMGHDpiGGYZgMh0cEDMMwGQ6PCBiGYTIcVgQMwzAZDisChmGYDIcVAcMwTIbDioBhGCbDYUXAMAyT4fw/YeGrw3QyCiMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fee0c3d50f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Training the model\n",
    "from dqn_agent import Agent\n",
    "\n",
    "agent = Agent(state_size=37, action_size=4, seed=0)\n",
    "#print([p for p in agent.qnetwork_local.parameters()])\n",
    "\n",
    "# #debug\n",
    "# state = env.reset()\n",
    "# print(agent.qnetwork_local(state))\n",
    "# print(agent.qnetwork_local.forward(state))\n",
    "# #end debug\n",
    "\n",
    "filename_to_save = 'ddqn_checkpoint.pth'\n",
    "final_eps = 0.01\n",
    "def dqn(n_episodes=2000, max_t=1000, eps_start=1.0, eps_end=final_eps, eps_decay=0.995):\n",
    "    \"\"\"Deep Q-Learning.\n",
    "    \n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode\n",
    "        eps_start (float): starting value of epsilon, for epsilon-greedy action selection\n",
    "        eps_end (float): minimum value of epsilon\n",
    "        eps_decay (float): multiplicative factor (per episode) for decreasing epsilon\n",
    "    \"\"\"\n",
    "    scores = []                        # list containing scores from each episode\n",
    "    scores_window = deque(maxlen=100)  # last 100 scores\n",
    "    eps = eps_start                    # initialize epsilon\n",
    "    for i_episode in range(1, n_episodes+1):     \n",
    "        env_info = env.reset(train_mode=True)[brain_name] # reset the environment\n",
    "        state = env_info.vector_observations[0]            # get the current state\n",
    "        score = 0   # initialize the score\n",
    "        for t in range(max_t): #this could also be while True instead\n",
    "            action = agent.act(state, eps)        # select an action\n",
    "            env_info = env.step(action)[brain_name]        # send the action to the environment\n",
    "            next_state = env_info.vector_observations[0]   # get the next state\n",
    "            reward = env_info.rewards[0]                   # get the reward\n",
    "            done = env_info.local_done[0]                  # see if episode has finished\n",
    "            agent.step(state, action, reward, next_state, done) #do the learning\n",
    "\n",
    "            score += reward                                # update the score\n",
    "            state = next_state                             # roll over the state to next time step\n",
    "            if done:                                       # exit loop if episode finished\n",
    "                break\n",
    "        scores_window.append(score)       # save most recent score\n",
    "        scores.append(score)              # save most recent score\n",
    "        eps = max(eps_end, eps_decay*eps) # decrease epsilon\n",
    "        print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)), end=\"\")\n",
    "        if i_episode % 100 == 0:\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))\n",
    "        if np.mean(scores_window)>=25.0:\n",
    "            print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode-100, np.mean(scores_window)))\n",
    "            torch.save(agent.qnetwork_local.state_dict(), filename_to_save)\n",
    "            break\n",
    "    torch.save(agent.qnetwork_local.state_dict(), filename_to_save) #for debug only\n",
    "    return scores\n",
    "\n",
    "strt = time.time()\n",
    "scores = dqn()\n",
    "print('Training Time is {}'.format(time.time()-strt))\n",
    "\n",
    "# plot the scores\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(len(scores)), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1: 16.0\n",
      "Episode 2: 19.0\n",
      "Episode 3: 13.0\n",
      "All the scores[16.0, 19.0, 13.0]\n",
      "Mean Score: 16.0\n"
     ]
    }
   ],
   "source": [
    "#Testing the model\n",
    "from dqn_agent import Agent\n",
    "filename_to_load = './ddqn_checkpoint.pth'\n",
    "final_eps = 0.01\n",
    "\n",
    "agent = Agent(state_size=37, action_size=4, seed=0)\n",
    "# load the weights from file\n",
    "#agent.qnetwork_local.load_state_dict(torch.load(filename_to_load))\n",
    "\n",
    "#Since the model is trained on gpu, need to load all gpu tensors to cpu:\n",
    "agent.qnetwork_local.load_state_dict(torch.load(filename_to_load, map_location=lambda storage, loc: storage))\n",
    "\n",
    "\n",
    "#print([p for p in agent.qnetwork_local.parameters()])\n",
    "\n",
    "# #debug\n",
    "# state = env.reset()\n",
    "# print(agent.qnetwork_local(state))\n",
    "# print(agent.qnetwork_local.forward(state))\n",
    "# #end debug\n",
    "\n",
    "num_episodes = 100\n",
    "scores = []\n",
    "for i_episode in range(1,num_episodes+1):\n",
    "    env_info = env.reset(train_mode=False)[brain_name] # reset the environment\n",
    "    state = env_info.vector_observations[0]            # get the current state\n",
    "    score = 0                                          # initialize the score\n",
    "    while True:\n",
    "        action = agent.act(state, eps=final_eps)        # select an action\n",
    "        env_info = env.step(action)[brain_name]        # send the action to the environment\n",
    "        next_state = env_info.vector_observations[0]   # get the next state\n",
    "        reward = env_info.rewards[0]                   # get the reward\n",
    "        done = env_info.local_done[0]                  # see if episode has finished\n",
    "        #agent.step(state, action, reward, next_state, done) #do the learning\n",
    "\n",
    "        score += reward                                # update the score\n",
    "        state = next_state                             # roll over the state to next time step\n",
    "        if done:                                       # exit loop if episode finished\n",
    "            print('Episode {}: {}'. format(i_episode, score))\n",
    "            scores.append(score)\n",
    "            break\n",
    "\n",
    "print('All the scores{}'.format(scores))\n",
    "print(\"Mean Score: {}\".format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
